Hereinafter, an outline of the following two technologies related to exposure control processing will be described as background technologies.
(1) A focal-plane shutter operation and occurrence of distortion
(2) Dynamic range expansion processing by exposure time control (shutter control)
(1) A Focal-Plane Shutter Operation and Occurrence of Distortion
First, a focal-plane shutter operation and occurrence of distortion will be described.
As a shutter operation system of image photographing processing in an image pickup apparatus, there is a shutter operation that controls exposure start and exposure end from one direction of an image pickup device surface. This shutter operation is called a focal-plane shutter operation or a rolling shutter operation. For example, there is a characteristic such that the exposure times between rows are slightly shifted when the exposure start and the exposure end are controlled from an upper row to a lower row of the image pickup device.
A configuration of a CMOS image sensor and an example of photographing processing will be described as an example of the image pickup device having a focal-plane shutter operation function with reference to FIG. 1.
FIG. 1 is a diagram illustrating apart of a configuration of an image pickup device (CMOS image sensor) 101. The image pickup device (CMOS image sensor) 101 is configured from a vertical scanning circuit 102, a horizontal scanning circuit 103, and a plurality of pixels 104 arranged in an array manner.
Electric charges are accumulated in a photodiode inside a pixel 104 by exposure processing associated with photographing of an object.
The electric charges accumulated in the photodiode of each pixel are output to a vertical signal line 113 through an amplifier transistor and a transfer transistor. A signal current output to the vertical signal line 113 is further supplied to the horizontal scanning circuit 103, is subjected to predetermined signal processing, and is then output outside through a signal output line 114.
Since vertically aligned pixels are commonly connected to the vertical signal line 113, to independently read out a signal of each pixel, only a signal of one pixel at a time should be output to the vertical signal line 113.
That is, in the image pickup device (CMOS image sensor) 101, a signal is read out from each of pixels 104d aligned in the lowermost row, first, as illustrated in FIG. 2(A). The readout of signals from the row of pixels 104c is performed, next, as illustrated in FIG. 2(B). Then, the readout of signals by changing the row is performed in sequence, so that the signal of each pixel can be independently read out. A control signal of the readout of pixels is, for example, output from a horizontal reset line 111 and a horizontal select line 112 connected to the vertical scanning circuit 102 illustrated in FIG. 1.
Each of the pixels 104 that configure the image pickup device (CMOS image sensor) starts exposure again immediately after the readout processing of the accumulated electric charges. That is, the exposure processing for a next image frame is started.
In this way, when the readout processing is executed in sequence in units of rows, and the exposure processing is started immediately afterwards, start times and end times of exposure of the photodiode 104a in the head row and the photodiode 104d in the bottom row are different, that is, a gap occurs between exposure times (or exposure periods).
This is a characteristic of the shutter operation, called a focal-plane shutter operation or a rolling shutter operation.
Note that, although only the four rows of 104a to 104d are illustrated in the drawing, this illustrates only a part of the image pickup device. A large number of rows, such as several hundred to several thousands of rows, are set in an actual image pickup device, and the readout is executed in a unit of each row in sequence.
An example of start and end timings of exposure of each row, and a start timing of readout of the electric charges will be described with reference to FIGS. 3 and 4.
In both of FIGS. 3 and 4, the horizontal axis represents a time axis and the vertical axis represents a row.
For example, in FIG. 3, a time gap occurs in the readout timings of the electric charges in units of rows, as illustrated in the drawing with the dotted lines 151a, a, and 151b. 
A plurality of rectangular blocks illustrated in FIG. 3 illustrates an exposure time of a given photographed image frame, and is an exposure time in units of row blocks made of a row or a plurality of rows.
Immediately after the timing illustrated in the readout line 151a of an image frame photographed in advance, the exposure processing is started. As illustrated in the readout line 151a, the exposure start times are slightly shifted in units of rows. In the graph illustrated in the drawing, an upper side row is started to expose in advance, and rows toward a lower side is started to expose later.
The uppermost row has the exposure start time at a time (t1), and the bottommost row has the exposure start time at a time (t2).
The right end of the plurality of rectangular blocks illustrated in FIG. 3 is a timing at which the readout processing of an exposed image is executed, and accumulated electric charges of pixels in each row is read out at a timing illustrated in the readout line 151b. 
In this example, the exposure end time≈the readout processing time, and as illustrated in the readout line 151b of FIG. 3, the readout processing of pixels in each row is performed from a head row in sequence.
The uppermost row has the exposure end time at a time (t2), and the bottommost row has the exposure end time at a time (t3).
Note that, in this example, since the exposure starts and the exposure ends of each row have gaps at the same timing in units of rows, the exposure times of all rows are the same.
FIG. 4 illustrates exposure processing and readout timing corresponding to images of two consecutive photographed frames at the time of photographing a moving image.
As illustrated in FIG. 4, a preceding photographed frame N has the exposure time during a period of readout lines 152a to 152b, and the readout of pixel values from each row is executed at the timing illustrated in the readout line 152b. 
A subsequent photographed frame N+1 has the exposure time during readout lines 152b to 152c, and the readout of pixel values from each row is executed at the timing illustrated in the readout line 152c. 
In the example illustrated in FIG. 4, the preceding photographed frame N has:
the exposure start time, in which the uppermost row has a time (t1a), and the bottommost row has a time (t1b), and
the exposure end time, in which the uppermost row has a time (t1b), and the bottommost row has a time (t1c).
The subsequent photographed frame N+1 has:
the exposure start time, in which the uppermost row has a time (t2a), and the bottommost row has a time (t2b), and
the exposure end time, in which the uppermost row has a time (t2b), and the bottommost row has a time (t2c).
In the example illustrated in FIG. 4, for example, the exposure time of the bottommost row of the preceding photographed frame N and the exposure time of the uppermost row of the subsequent photographed frame N+1 are almost the same period of time. That is, a phenomenon occurs, in which lower side image data of the preceding image frame and upper side image data of the subsequent frame are images at almost the same period of time.
As a result, for example, when a moving object is photographed, or when photographing processing is performed, in which the camera itself is moved during exposure, distortion occurs in an image due to the gap of exposure times between rows caused by a focal-plane shutter operation.
An example of image distortion will be described with reference to FIG. 5.
FIG. 5(A) is a photograph example photographed with the camera being stopped.
FIG. 5(B) is a photograph example photographed while the camera is moved in a lateral direction.
While distortion does not occur in the image of FIG. 5(A), distortion occurs in the image illustrated in FIG. 5(B).
Similarly, FIG. 5(C) is a photograph example when a car is photographed with being stopped.
FIG. 5(D) is a photograph example when a car is photographed with being moved.
While distortion does not occur in the image of FIG. 5(C), distortion occurs in the image illustrated in FIG. 5(D).
Such distortion occurs because there is a gap between the exposure times of the image pickup device described with reference to FIGS. 3 and 4, that is, the exposure times are slightly different from an upper end row to a lower end row. This occurrence phenomenon of distortion is referred to as a focal-plane shutter phenomenon, or a rolling shutter phenomenon.
The image pickup device that performs a focal-plane shutter operation is capable of obtaining a photographed image without distortion or having reduced distortion by mechanically shading light reaching the image pickup device.
A digital camera that employs a mechanical focal-plane shutter (hereinafter, a mechanical shutter) used in a silver halide camera has been known. That is, the digital camera has a configuration in which a front curtain that controls exposure start and a back curtain that controls exposure end are performed by mechanical shutters as dousers. The widths of these two curtains are adjusted to control the exposure time. The speed of the mechanical shutter is higher than the focal-plane shutter (readout scanning of the sensor) of the image pickup device, and a photographed image having reduced distortion can be obtained.
Patent Documents 1 and 2 disclose a configuration using an electronic shutter as the front curtain that controls the exposure start and using a mechanical shutter (douser) as the back curtain that controls the exposure end. The electronic shutter that is the front curtain is performed by reset scanning of pixels in time with travelling of the mechanical shutter that is the back curtain. That is, it is a method of reading out a signal after the mechanical shutter that is the back curtain is closed and the pixels are shaded.
Patent Document 1 discloses a configuration in which only the electronic shutter is used in the exposure start and the exposure end when a moving image is photographed, and the exposure start is controlled by the electronic shutter and the exposure end is controlled by the mechanical shutter when a still image is photographed. This configuration enables the photographing of a moving image as well as a sufficiently accurate shutter operation even when a still image is photographed.
Patent Document 2 describes an effect of reducing a mechanical shutter that controls the exposure start by controlling the exposure start by the electronic shutter and controlling the exposure end by the mechanical shutter.
Patent Documents 1 and 2 only disclose a configuration in which the exposure time, that is, the time from start to end of exposure is the same length in all pixels.
(2) Dynamic Range Expansion Processing by Exposure Time Control (Shutter Control)
Next, dynamic range expansion processing by exposure time control (shutter control) will be described.
A dynamic range of a photographed image can be expanded by controlling the exposure time to each pixel of the image pickup device.
In a bright object region, if the exposure time is made longer, the accumulated electric charges of a pixel are saturated, and an accurate pixel value cannot be obtained.
Meanwhile, in a dark object region, if the exposure time is made longer, an accurate pixel value corresponding to object luminance can be easily obtained.
Therefore, in the region where the object is bright, a pixel value of a pixel, in which the exposure time is set short, is obtained as an effective pixel value. Meanwhile, in the region where an object is dark, a pixel value of a pixel, in which the exposure time is set longer, is obtained as an effective pixel value. A method of expanding the dynamic range is to combine these values to generate an output image. Note that, in outputting a final pixel value, pixel value adjustment processing based on the exposure time is executed.
Patent Documents 3 and 4, and Non Patent Document 1 disclose a technology in which a different exposure time is set in each pixel row of the image pickup device, and the dynamic range of a photographed image is expanded. For example, it is configured such that a short time exposure row and a long time exposure row are set in every other pixel row of the image pickup device.
For example, Patent Document 3 discloses a configuration in which a high sensitive pixel (long time exposure pixel) and a low sensitive pixel (short time exposure pixel) are set by causing the electronic shutter operation of the CMOS image sensor to perform different operations in even rows and in an odd rows. A high dynamic range image can be photographed by combination of pixel values according to object luminance.
Patent Document 4 discloses a configuration in which two patterns of exposure times are set in each row, or in each of a plurality of rows with more than one row, and the patterns are changed by an electronic shutter operation. Further, Patent Document 4 also discloses a method in which three patterns of exposure times are set, and are set in units of rows according to the brightness of a portion of a screen.
Further, Non Patent Document 1 discloses a configuration in which arbitrary exposure time control can be further freely performed in units of rows. The technique of Non Patent Document 1 is a configuration in which a dynamic range can be expanded by setting an individual exposure time in units of rows according to the brightness of an image.
The configurations of Patent Documents 3 and 4, and Non Patent Document 1 execute the exposure time control in units of regions by a configuration using an electronic shutter.
Further, Patent Document 5 discloses a configuration, in which processing that executes reset scanning twice in every one row in an alternate manner, the reset scanning once resetting the electric charges accumulated in each pixel by the electronic shutter, or multiple reset processing that executes the reset processing three times at the rate of once in every three rows is performed, and the exposure control by a configuration of a mechanical shutter and an electronic shutter used in combination is performed, so that the photographing with a plurality of predetermined different exposure times in units of rows is performed, and the dynamic range is expanded.
However, Patent Document 5 described above has a configuration in which a plurality of different exposure times is set in units of rows in advance and the photographing is performed, and does not have a configuration in which adaptive exposure time control is performed according to the luminance of an object.
(3) Summary of Related Art
As described above, in the configuration using a focal-plane shutter, a gap of an exposure period occurs in units of rows, and distortion caused by the gap occurs.
Further, while the technology of expanding the dynamic range by performing exposure period control in units of regions has been known, even in this configuration, a gap of an exposure period in units of rows cannot be prevented when a focal-plane shutter is used, and distortion caused by the gap occurs.